Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistent store and forward #1680

Merged
merged 1 commit into from
Apr 6, 2020
Merged

Persistent store and forward #1680

merged 1 commit into from
Apr 6, 2020

Conversation

sdbondi
Copy link
Member

@sdbondi sdbondi commented Apr 6, 2020 •
edited
Loading

Description

  • Implemented sqlite persistence for store and forward (SAF).
  • Nodes keep track of a timestamp that they last requested messages for
  • Nodes use that timestamp when requesting messages to reduce the number
    of duplicate messages requested
  • Request for SAF messages is broken up into 3 responses: Discovery, Join
    and ExplicitlyAddressed - respectively, discovery messages, join messages and
    messages that are explicitly addressed to this node.
  • DB migrations run on node startup
  • Sqlite operations run on tokio blocking threads
  • Sqlite connection interface that allows usage of sqlite's in-memory
    database connection in addition to the file-system database

TODO:

  • Verify that the node requested the store and forward messages before accepting them using the nonce already included in the request and response messages.
  • Clean out stored messages when they have been delivered

Motivation and Context

In order for store and forward to be reliable it will need to be persistent between node restarts and be able to store and fetch thousands of messages.

How Has This Been Tested?

Some unit testing, memorynet store and forward test works. memorynet takes a wallet offline before another peer attempts to discover it, waits a while for the discovery message to propagate through the network, brings the peer online before the discovery times out, requests SAF messages and see if discovery succeeds.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • [~] Breaking change (fix or feature that would cause existing functionality to change)
  • Feature refactor (No new feature or functional changes, but performance or technical debt improvements)
  • New Tests
  • Documentation

Checklist:

  • I'm merging against the development branch.
  • I ran cargo-fmt --all before pushing.
  • I have squashed my commits into a single commit.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

@sdbondi sdbondi force-pushed the sb-dht-storage branch 5 times, most recently from b1a19f7 to a42c5ce Compare April 6, 2020 15:06
@sdbondi
Persistent store and forward
45a7b21 
- Implemented sqlite persistence for store and forward (SAF).
- Nodes keep track of a timestamp that they last requested messages for
- Nodes use that timestamp when requesting messages to reduce the number
  of duplicate messages requested
- Request for SAF messages is broken up into 3 responses: Discovery, Join
  and ExplicitlyAddressed - respectively, discovery messages, join messages and
  messages that are explicitly addressed to this node.
- DB migrations run on node startup
- Sqlite operations run on tokio blocking threads
- Sqlite connection interface that allows usage of sqlite's in-memory
  database connection in addition to the file-system database
CjS77
CjS77 approved these changes Apr 6, 2020
View reviewed changes
Copy link
Collaborator

@CjS77 CjS77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CjS77 CjS77 merged commit 3c18269 into development Apr 6, 2020
@CjS77 CjS77 deleted the sb-dht-storage branch April 7, 2020 10:00
SWvheerden added a commit that referenced this pull request Mar 20, 2024
@hansieodendaal @SWvheerden
feat: cache new block template (#6222)
e0ad342 
Description
---
In the merge mining proxy for the block template protocol:
- Cached a new block template and used it in case it is asked
repetitively for the same best block.
- Added an exit clause to the potential endless loop.

Motivation and Context
---
Under certain conditions, a merge mining proxy request for a new block
template from the base node would be repeated many times for the same
best block height, without the final request for a block template that
includes the coinbase transaction. This wastes many resources,
especially within the mempool where the template must be constructed.
```
2024-03-15 16:49:43.873749100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1675
2024-03-15 16:56:35.702267000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1676
2024-03-15 16:56:57.709323100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1677
2024-03-15 16:59:04.747989700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.840957100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.849282000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.852855000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.953873500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.954461600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.958619200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.054517300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.056071600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.057842800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.058706200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.184297100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.184568300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.186171600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.308025900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.308188600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.411993800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.413148500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.436028800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.457553200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.569510200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.573500600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.658578700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.659258900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.693116700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.695350400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.713246600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.724627700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.734541000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.904546600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.909569300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.913773800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.914761900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.918061100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.928847000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.125242100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.125427400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.130678700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.367898700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.385011900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.386722300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.388034000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.406540100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.645604600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.654252400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.659640400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.661591500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.829952300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.838337400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.843845500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.847976400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.036353600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.038593100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.042252500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.070333200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.204174900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.207776400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.208302900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.211634900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.218055300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.367296100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.367378600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.369022900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.372211400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.504687000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.505760400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.505823500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:09.578821600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:09.829455600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 17:00:11.282291600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1679
2024-03-15 17:01:13.476999400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1680
```

How Has This Been Tested?
---
System-level testing

What process can a PR reviewer use to test or verify this change?
---
- Review code changes
- System-level testing:
- These log entries should not be repeated for the same best block
height:
  ```
2024-03-18 15:12:50.362269600
[minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new
block template at height: #1832 (try 4)
  ```
  - Look for these log entries instead:
  ```
2024-03-18 15:13:46.016447400
[minotari_mm_proxy::proxy::block_template_protocol] DEBUG Used existing
new block template at height: #1836 (try 1)
  ```

<!-- Checklist -->
<!-- 1. Is the title of your PR in the form that would make nice release
notes? The title, excluding the conventional commit
tag, will be included exactly as is in the CHANGELOG, so please think
about it carefully. -->


Breaking Changes
---

- [x] None
- [ ] Requires data directory on base node to be deleted
- [ ] Requires hard fork
- [ ] Other - Please specify

<!-- Does this include a breaking change? If so, include this line as a
footer -->
<!-- BREAKING CHANGE: Description what the user should do, e.g. delete a
database, resync the chain -->

---------

Co-authored-by: SW van Heerden <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CjS77 CjS77 CjS77 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sdbondi @CjS77